Skip to content

[REST API] Track assigned A/B test variant #8753

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 5 commits into from
Jan 27, 2023
Merged

[REST API] Track assigned A/B test variant #8753

merged 5 commits into from
Jan 27, 2023

Conversation

@selanthiraiyan
Copy link
Contributor

@selanthiraiyan selanthiraiyan commented Jan 26, 2023 •
edited
Loading

Part of: #8679

Description

This PR adds tracking for the REST API A/B test. This lets us track the variant assigned to the user.
Internal ref - pe5sF9-13h-p2

Changes

  • Added track event to track the assigned variant.
  • Unit test to validate tracking.
  • Refactor unit tests.

Testing instructions

  • Log out of the app
  • Launch the app and skip onboarding if needed.
  • From the prologue screen you can know which variant has been assigned to your device.
  • Control will have the existing login screen has both WPCOM and store address login options.
  • Treatment will only have the store address login option.
  • Check Xcode logs and ensure that the following event is tracked.
Tracked rest_api_login_experiment, properties: [AnyHashable("experiment_variant"): "control"]
  • Value for experiment_variant will vary based on the assigned variant. Currently, the experiment is staging, so we will always get the value control` here. 21039-explat-experiment

Screenshots

Control Treatment
Prologue screen

Tracks event registration

https://github.com/Automattic/tracks-events-registration/pull/1355

  • I have considered if this change warrants user-facing release notes and have added them to RELEASE-NOTES.txt if necessary.

@selanthiraiyan selanthiraiyan added type: task An internally driven task. feature: REST API Authenticating requests using application password and using REST API instead of Jetpack tunnel. labels Jan 26, 2023
@selanthiraiyan selanthiraiyan added this to the 12.1 milestone Jan 26, 2023
@selanthiraiyan selanthiraiyan mentioned this pull request Jan 26, 2023
Closed
6 tasks
@selanthiraiyan selanthiraiyan added the category: tracks Related to analytics, including Tracks Events. label Jan 26, 2023
@wpmobilebot
Copy link
Collaborator

wpmobilebot commented Jan 26, 2023 •
edited
Loading

You can test the changes from this Pull Request by:
  • Clicking here or scanning the QR code below to access App Center
  • Then installing the build number pr8753-bc70ac9 on your iPhone

If you need access to App Center, please ask a maintainer to add you.

@selanthiraiyan selanthiraiyan marked this pull request as ready for review January 26, 2023 08:37
@pmusolino pmusolino self-assigned this Jan 26, 2023
pmusolino
pmusolino reviewed Jan 26, 2023
View reviewed changes
Copy link
Contributor

@pmusolino pmusolino left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It could be just an unlucky case, but I always get control as an experiment, I tried multiple times (also erasing the content on the simulator) but I wasn't able to get the Treatment. 😢
What it could be? I imagine that on around 20 tests I should see a treatment at least.

@selanthiraiyan
Copy link
Contributor Author

selanthiraiyan commented Jan 27, 2023 •
edited
Loading

It could be just an unlucky case, but I always get control as an experiment, I tried multiple times (also erasing the content on the simulator) but I wasn't able to get the Treatment. 😢
What it could be? I imagine that on around 20 tests I should see a treatment at least.

I have configured 50% - 50% for control and treatment. I think we will always get the control variant until the experiment starts running. I have updated the PR description to include this detail.

@selanthiraiyan selanthiraiyan force-pushed the feat/8679-rest-api-track-ab-variant branch from 9176e6c to bc70ac9 Compare January 27, 2023 08:19
@peril-woocommerce
Copy link

peril-woocommerce bot commented Jan 27, 2023 •
edited
Loading

Warnings
⚠️ This PR is assigned to a milestone which is closing in less than 2 days Please, make sure to get it merged by then or assign it to a later expiring milestone

Generated by 🚫 dangerJS

@selanthiraiyan
Copy link
Contributor Author

selanthiraiyan commented Jan 27, 2023 •
edited
Loading

@pmusolino

I have updated the abacus experiment to a newer version. (In the PR's base branch)

I made this change to address the feedback I received on the experiment proposal. pbxNRc-2i4-p2 I have validated this new experiment with my sandbox.

This is FYI, and this experiment version change won't affect this analytics tracking PR.

pmusolino
pmusolino approved these changes Jan 27, 2023
View reviewed changes
Copy link
Contributor

@pmusolino pmusolino left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Go for it, it seems that unless the experiment is online we'll get always control :shipit:

Base automatically changed from feat/8679-replace-rest-api-feature-flag-with-ab-test to trunk January 27, 2023 11:29
@selanthiraiyan selanthiraiyan merged commit 37a913b into trunk Jan 27, 2023
@selanthiraiyan selanthiraiyan deleted the feat/8679-rest-api-track-ab-variant branch January 27, 2023 11:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@iamgabrielma iamgabrielma Awaiting requested review from iamgabrielma

1 more reviewer

@pmusolino pmusolino pmusolino approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@pmusolino pmusolino

Labels

category: tracks Related to analytics, including Tracks Events. feature: REST API Authenticating requests using application password and using REST API instead of Jetpack tunnel. type: task An internally driven task.

Projects

None yet

Milestone

12.1

Development

Successfully merging this pull request may close these issues.

4 participants

@selanthiraiyan @wpmobilebot @pmusolino